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ABSTRACT 



When analyzing open-ended or categorical questions, many 
times responses are cross-classified by other categorical variables. The 
resulting contingency tables are then analyzed using the chi square test of 
independence. This procedure leads to multiple significance tests and 
provides no method to assess higher order interactions. Loglinear modeling, 
however, permits the user to assess interaction effects as well as the 
effects of variables. This study compared the results of using these two 
procedures. Data from three open-ended questions (about program strengths, 
weaknesses, and suggested improvements) in a survey of high school 
participants in an interactive video program were analyzed by multiple chi 
square tests of independence and loglinear modeling. Sample size ranged from 
204 to 146 for the three questions. In two cases (strengths and suggested 
improvements) , the results were similar. The question about weaknesses, 
however, required a three-way interaction. Since higher order interactions 
are not possible with the chi square test of independence, it is concluded 
that loglinear modeling provides a more effective way of analyzing multi-way 
contingency tables. (Contains three figures.) (Author/SLD) 
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Abstract 

When analyzing open-ended or categorical questions, may times responses are cross-classified by 
other categorical variables. The resulting contingency tables are then analyzed using the x 2 (chi 
square) test of independence. This procedure leads to multiple significance tests and provides no 
method to assess higher order interactions. Loglinear modeling, however, permits the user to 
assess interaction effects as well as the effects of the variables. This study compared the results of 
using these two procedures. 

Data from three open-ended questions (Strengths, Weaknesses, Suggested Improvements) 
in a survey of high school participants in an interactive video program was submitted to analysis 
by multiple x 2 tests of independence and by loglinear modeling. Sample size ranged from 204 to 
146 for the three questions. In two instances (Strengths, Suggested Improvements), the results 
were similar. One question (Weaknesses), however, required a three-way interaction. Since 
higher order interactions are not possible with the x 2 test of independence, it was concluded that 
loglinear modeling provided a more effective method of analyzing multi-way contingency tables. 
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Chi-square or Log-Linear Modeling: Is there a difference? 

In analysis of open-ended questions, many times responses are summed to categories and 
the categories contrasted by chi-square (x 2 ) goodness of fit. Categories may then be cross- 
classified by another factor and independence of variables tested by the x 2 test of independence. 
Categories may again be cross-classified by a second factor and the process repeated. This 
procedure leads to multiple significance tests and provides no method to assess interaction of 
more than two variables. Log-linear modeling, on the other hand, permits the user to assess 
interaction between multiple variables. This study investigates this problem by comparing results 
and interpretation utilizing multiple chi-square tests of independence with those obtained using a 
log-linear model of the same data. 

Fienberg (1989) says that the use of multiple two-dimensional tables to analyze data fails 
to distinguish the relationship when other variables are present, does not permit simultaneous 
examination of the relationship, and ignores higher-order interactions (p. 1). In addition, the use 
of multiple tests of significance inflates the type I error rate. Since one goal of research is to 
determine what has occurred, contrasting two procedures using the same data enables the 
researcher to evaluate the effectiveness of these procedures thus making better analytic decisions 
in future research. 

Method 

All high school students enrolled in an educational interactive video class at a facility in 
Virginia during the Spring semester, 1996, were surveyed. Surveys were administered during the 
regularly scheduled class time by the class instructor or remote facilitator. Of the 238 returned 
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surveys, 77 respondents were participating from the remote site with 161 respondents at the home 
site. Since this is a relatively new program and is still expanding, respondents were also classified 
as those participating in a recently (1995/96) opened site and those participated in an established 
site. A total of 125 respondents were participating in first-year sites while 1 13 were at previously 
established sites. Surveys were returned for 12 classes (all levels of Spanish were coded Spanish) 
from 14 high schools. 

Measurement and Analysis 

The survey instrument consisted of demographic information (school, gender, grade, etc.) 
and 34 questions. Question 1 requested reason for taking the ITV (interactive video) class. 
Questions 2-31 were 5-point Likert style questions. Questions 32-34 were open-ended requesting 
strengths, weaknesses, and suggestions for improvement for the ITV program. Questions 21-26 
were to be answered by remote site participants only. Only questions 32-34 were used in this 
analysis. 

Responses to open-ended questions (Q32-34) were coded based on the response. After 
determining similarities of the responses, these were placed in categories. For example, the 
suggested improvements response “have schools on the same schedule” was coded as ‘38'. It and 
the responses “synchronize time schedule” and “establish snow schedule” were then summed to a 
major category “scheduling*. 

The questionnaire provided three blank lines after each heading (strengths, weaknesses, 
suggested improvements). Thus a respondent could contribute three responses to the scheduling 
category. To prevent a respondent from being counted twice (ie, having more responses than 
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respondents), only the strength, weakness, or suggested improvement listed first was included in 
this analysis. 

The independence of the strength response by site (home/remote) was tested using the x 2 
(chi-square) test of independence. Independence of strength response by year was also tested 
using the x 2 test of independence. Responses were then analyzed using a log-linear model. While 
still using the predetermined strength categories, site and year were entered as factors 
simultaneously with the strength variable. This procedure was repeated to analyze the weakness 
and suggested improvements variables. All analyses were conducted using a=.05. 

Results and Discussion 

Strengths 

Of the 238 respondents, 204 listed one or more strengths of the interactive video program. 
Responses to the open-ended strength question were summed to form five categories for the 
strength response: interaction with other participants, access to previously unavailable classes, 
learning, student behavior, and other (see Figure 1). Using the x 2 test of independence, there was 
a statistically significant relationship between site and strength (x^lO.82, df=4, p<03). Of the 65 
remote site respondents answering the strength question, 40% cited access to previously 
unavailable classes. Less than 20% of the 139 home site respondents referred to the same 
strength. Using the criteria of standardized residual of 2 or larger, remote she respondents cited 
access to class as a strength more frequently than expected. No other category contributed 
significantly to the omnibus x 2 A statistically significant association between year and strength 
was not detected ( j £= 2 . 1 9 df=4, p>.05). 
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Figure 1 : Strengths of the Interactive Video Program 
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When testing strength by site by year in a loglinear model, the model was adequately 
represented with no interaction terms (x 2 =l 8.6, df=13, £=. 13). There was, however, a significant 
improvement in fit (Ax 2= 10, Adf =4, p< 05) if an interaction between strength and site is added to 
the model (x^.52, d£=9, p=58). Both analyses, therefore, reach the same conclusion. There is 
a relationship between site and strength reported. Remote site respondents report access to 
previously unavailable classes as a strength in greater proportion than home site respondents. 
Weaknesses 

Of the 238 respondents, 186 submitted one or more replies concerning weaknesses of the 
interactive video program. Weakness responses were coded and summed to six categories: 
student behavior, instruction, equipment, scheduling, audio, and other (see Figure 2). A 
statistically significant relationship was detected between site and weakness (x 2= 29.79, df=5, 
p<.01). Using the standardized residual greater than two, remote site respondents report student 
behavior a weakness less frequently than expected. In addition, more remote site and less home 
site respondents report instruction as a weakness than expected. There was no statistically 
significant relationship between weakness and year of program existence. 

The loglinear model of weakness deteriorates significantly if a 3-way interaction of year, 
site, and weakness is not included (A x 2 ^ 15.5, Adf =5, p<.05). To assist interpretation of the 
three way interaction, the chi-square test of independence was again conducted between site and 
weakness while controlling for year of program existence. There was no relationship between site 
and weakness during the first year of program existence (x 2= 9. 1, df =5, p=. 10). During the 
second year of program existence, however, there was a statistically significant relationship 
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between site and weakness (x 2== 32, df =5, p< 01). Contributing to significance, fewer remote site 
respondents report student behavior (0%) a weakness than expected and more remote site 
respondents (39%) report instruction a weakness than expected. Fewer home site respondents 
(4%) report instruction a weakness than expected. When controlling for site, there was no 
relationship between year and weakness response at the remote site. There was a relationship 
between year and weakness response when controlling for home site (x^HAS, df =5, p<05). 

No cell, however, contained a standardized residual greater than 2. 

Suggested Improvement 

One hundred forty six respondents suggested improvements for the interactive video 
program. These suggested improvements were coded and summed to seven categories, 
equipment, interaction, instruction, scheduling, audio, variety of classes offered, and other (see 
Figure 3). A statistically significant relationship was detected between suggested improvement 
and site (x^fiJO, df=6, p<01) and between suggested improvement and year (x 2 =42.78, df=6, 
p<.01). Contributing to statistical significance by site were fewer remote site respondents 
suggested improvement of equipment than expected while more suggested improvement of 
instruction and scheduling. On the other hand, fewer home site respondents suggested 
improvement of instruction and scheduling than expected. Contributing to statistical significance 
by year, more respondents in the second year of program existence suggested improvement in 
equipment than expected while fewer than expected in the first year and more first year 
respondents suggested improvement in instruction than expected and fewer second year. 
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When using a loglinear model, two 2-way interactions were required to provide a non- 
significant fit (x 2 =12.48, df=7, p=. 08). The model thuS determined contained interactions 
between suggested improvement and site and between suggested improvement and year. This 
model corresponds to the analysis using the x 2 test of independence. 

Conclusion 

Although there was the expected similarity of results from the two procedures used, there 
were also some discrepancies. Analysis of the Strength variable by site and year yielded similar 
results in both analyses. Using the test of independence, a relationship between strength and site 
was detected, but no relationship between strength and year. The loglinear model yielded similar 
results. Using the test of independence, a relationship between suggested improvement and site 
and between suggested improvement and year was also detected. The loglinear model again 
yielded similar results. Using the test of independence, a relationship between weakness and site 
was detected, but no relationship between weakness and year. The loglinear model, however, 
required a three-way interaction between weakness, site, and year. This interaction can only be 
detected in an independence test by conditioning or controlling for one of the variables and noting 
a difference in responses. Although this study only used three variables in each condition, the 
effectiveness of a loglinear model was apparent. It was concluded, therefore, that loglinear 
modeling provides a more effective method of analyzing multi-way contingency tables. 
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